Skip to content

Cap a failed observation's error_message and reserve the openarmature_ namespace - #288

Merged
chris-colinsky merged 2 commits into
mainfrom
feature/0119-error-message-cap-and-reserved-keys
Sep 3, 2026
Merged

Cap a failed observation's error_message and reserve the openarmature_ namespace#288
chris-colinsky merged 2 commits into
mainfrom
feature/0119-error-message-cap-and-reserved-keys

Conversation

@chris-colinsky

@chris-colinsky chris-colinsky commented Aug 30, 2026

Copy link
Copy Markdown
Member

Implements proposal 0119, accepted at spec v0.116.0. It closes the two error-channel edges 0118 left open.

The cap

A failed observation's error_message is now subject to the §5.5.5 per-value byte cap on all four mapped provider observations (Generation, Embedding, Tool, Retriever). It was written verbatim before this: 0118 classified the field for gating without saying it was subject to truncation, so a provider returning a very large exception string (an echoed HTML error page) rendered in full.

It takes the contract's direct-application arm rather than its inheritance arm. The OTel surface defines no error_message span attribute, so the value arrives untruncated and the observer that writes it applies its own payload_byte_cap, rather than inheriting a cap from an upstream OTel truncation that never happened. Re-applying a second cap to an already-truncated value would move the marker and misreport its byte total, which is why the two arms source the cap differently.

Both surfaces of the Tool arm are capped. A failed Tool observation renders the same harvested string twice, once in metadata.error_message and once as the observation's statusMessage. §5.5.5 governs payload-classified values rather than payload-classified fields, so capping one while the other still carried the whole exception would defeat the cap. Spec confirmed this reading after the work was done: an implementation that caps only the metadata copy is non-conforming, and §8.7 gains a sentence naming statusMessage as the case in point. The remaining statusMessage writes take the error category, a classification token, and stay uncapped.

The omission arm is unchanged. §6's prohibition on surfacing the message through statusMessage is scoped to substituting for an omitted message, so a withheld message still leaves statusMessage null on a Tool failure. Under the default posture (disable_provider_payload=True) the field is absent entirely per 0118, so the cap is observable only where payloads are enabled.

The reserved keys

openarmature_ joins openarmature. and gen_ai. as a reserved caller-metadata namespace prefix, and four exact names (error_type, error_message, token_budget, token_budget_exceeded) join the reserved set, which grows from 29 to 33.

With error_message absent under the default posture, an unreserved caller key of that name would otherwise land unopposed in the very field 0118 requires to be absent, reintroducing through the metadata channel the leak the gate closes.

Pre-1.0 behavioral change: a caller passing invocation_metadata with a key beginning openarmature_, or with any of those four names, is now rejected at the invoke() boundary with ValueError where it previously passed.

Testing

Seven cases across six tests, each mutation-verified. Eight mutants run, all killed.

Two of them mattered. Reverting the cap at each of the four sites individually left the embedding and rerank arms green: the first pass covered only the LLM and Tool handlers, and a passthrough mutation of the shared helper (which breaks all four sites at once) produced exactly two failures rather than four. The suite would have accepted a two-arm implementation of a four-arm rule. Both arms now have tests, each verified to kill only its own case, and the passthrough mutant now trips all four.

Ahead of the pin

Spec v0.116.0 is beyond the current v0.112.0 pin, so the behavior ships unit-tested and the conformance.toml entry and fixtures ride the pin bump, matching the precedent in this cycle.

No fixture can drive the Tool arm today: the conformance adapter exposes no tool-calling node, so that arm is unit-tested only. Reported to spec rather than filing a fixture that would be deferred on arrival.

Correction (2026-09-03): the struck sentence is wrong. The conformance adapter has driven tool calls since 0092: calls_tool with a mock_tool block is a supported directive, mock_tool: {raises: {error_type, message}} produces a ToolCallFailedEvent carrying the fixture's own message, and fixture 098 case 2 already drives exactly that into a Langfuse Tool observation and asserts error_message in its metadata. It runs green. The real gap is narrower: fixture 160 carries no Tool case, and one is writable today against existing machinery. Corrected in the CHANGELOG and retracted to spec. Fixture 160's other five cases span Embedding, Generation, and Retriever, and would have caught the two-arm gap independently; they land at the pin bump.

Implements proposal 0119, accepted at spec v0.116.0. Two halves.

The cap. A failed observation's error_message now goes through the
section 5.5.5 per-value byte cap at all four mapped provider
observations. It takes the direct-application arm rather than the
inheritance arm: the OTel surface defines no error_message span
attribute, so the value arrives untruncated and the observer that
writes it applies its own payload_byte_cap. Re-applying a second cap to
an already-truncated value would move the marker and misreport its byte
total, which is why the two arms source the cap differently.

A failed Tool observation renders the same harvested string twice, in
metadata.error_message and as the observation's status message, and
both copies are capped. Section 5.5.5 governs payload-classified values
rather than payload-classified fields, so capping one surface while the
other carried the whole exception would defeat the cap. The remaining
status message writes take the error category, a classification token,
and stay uncapped. The omission arm is unchanged: a withheld message
still leaves the status message null on a Tool failure.

The reserved keys. openarmature_ joins openarmature. and gen_ai. as a
reserved caller-metadata namespace prefix, and four exact names join
the reserved set, taking it from 29 to 33. With error_message absent
under the default posture, an unreserved caller key of that name would
otherwise land unopposed in the very field 0118 requires to be absent.
This is a pre-1.0 behavioral change: a caller passing invocation
metadata with such a key is now rejected at the invoke boundary.

Spec v0.116.0 is beyond the current v0.112.0 pin, so the behavior ships
ahead of the pin, unit-tested; the conformance entry and fixtures ride
the pin bump. No fixture can drive the Tool arm today because the
conformance adapter exposes no tool-calling node.
Copilot AI lite review requested due to automatic review settings August 30, 2026 21:37

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Implements accepted spec proposal 0119 (v0.116.0) in the Python implementation by (1) enforcing the per-value byte cap on failed-observation error_message in the Langfuse mapping and (2) expanding reserved caller invocation-metadata keys to prevent collisions/leaks under the default privacy posture.

Changes:

  • Cap error_message emission in the Langfuse observer (LLM, tool, embedding, rerank), including the tool arm’s duplicated statusMessage surface.
  • Reserve the openarmature_ invocation-metadata namespace prefix and add new reserved exact keys (error_type, error_message, token_budget, token_budget_exceeded).
  • Add focused unit tests and update docs/changelog to reflect the new behavior.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/unit/test_observability_metadata.py Adds unit tests asserting openarmature_ prefix and new reserved exact keys are rejected at validation.
tests/unit/test_observability_langfuse.py Adds unit tests asserting error_message truncation behavior across observer arms and tool status_message.
src/openarmature/observability/metadata.py Expands reserved prefixes and reserved key names for invocation metadata validation.
src/openarmature/observability/langfuse/observer.py Applies payload_byte_cap truncation to error_message emission, including tool status-message duplication.
src/openarmature/AGENTS.md Updates agent-facing guidance to reflect capped error_message behavior under Langfuse payload settings.
docs/concepts/observability.md Updates observability docs to describe capped error_message behavior and tool dual-surface capping.
docs/agent/non-obvious-shapes.md Mirrors the Langfuse isolation/payload guidance updates for agent documentation.
CHANGELOG.md Adds a release note entry describing the cap and reserved-namespace behavior (ships ahead of the spec pin).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/openarmature/observability/metadata.py
Ten findings. The two largest were one defect seen from both sides.

0119 requires a repository check that fails when a top-level metadata
key written by the section 8 mapping is neither reserved by name nor
covered by a reserved namespace, and it was not built. It is now. It
discovers the metadata bags from what is passed to a metadata argument
rather than naming one: the observer builds five differently-named bags,
so a check hardcoded to the obvious one would have scanned one of five
and reported clean. That is the same subset failure the proposal warns
about, one level down.

The check finds three uncovered keys, all on the failure-isolation
marker span, where caller metadata is merged last with no collision
check so a caller key wins. They are held in a named exclusion set
rather than reserved unilaterally: that span is a graph mechanism no
mapping table covers, so whether the reserved set should reach an
unmapped span is a spec question. A new unreserved key still fails the
check, and a paired assertion fails if a held key stops being emitted so
the exclusion cannot outlive its subject.

The cap tests were vacuous. They asserted the value got shorter, not
that it was truncated by the specified algorithm, so a marker-less byte
chop passed all of them. They now pin the marker, its pre-truncation
byte total, and that the kept bytes prefix the original. A multibyte
case covers the code-point backtracking, which no test reached: at a
256-byte cap the cut lands on a 4-byte boundary and the loop is dead
code, so the test uses 257.

Truncation is now surrogate-safe. The cap runs on the failure path
before the observation is created, and an observer that raises is only
warned about, so an unguarded encode of harvested exception text would
have deleted the observation reporting the failure.

Also: reserved-key docs brought in sync across three sites and the
module docstring, the prefix rejection message built from the tuple
instead of a stale hand-copied list, a spec reference moved out of a
docstring, a comment this change made stale removed, and test matchers
tightened onto the rule rather than the echoed key.
@chris-colinsky
chris-colinsky merged commit 3f4e2d7 into main Sep 3, 2026
6 checks passed
@chris-colinsky
chris-colinsky deleted the feature/0119-error-message-cap-and-reserved-keys branch September 3, 2026 05:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants